Fitting annotation models to crowd-sourced measurements of populism in textual data

Estimation

I obtain MCMC estimates using JAGS with three chains, 5K burn-in iterations, and 40K iterations with thinning parameter set to 20. These choices are based on inspecting convergence and autocorrelation in initial models with fewer iterations and less (or no) thinning.

# subset codings
people_codings <- codings %>%
  mutate(judgment = as.integer(people)) %>% 
  filter(!is.na(judgment)) %>% 
  select(index, item, coder, judgment) 

# contruct model-compatible MCMC data object
people_mcmc_data <- get_codings_data(people_codings)

# initialization values
init_vals <- lapply(1:n_chains, function(chain) {
  
  out <- list()
  out[["pi"]] <- .2 + rnorm(1, 0, .05)
  out[[".RNG.name"]] <- "base::Wichmann-Hill"
  out[[".RNG.seed"]] <- 1234
  
  return(out)
})

# initilaize model
people_mcmc_model <- jags.model(
  file = model_file_path
  , data = people_mcmc_data
  , inits = init_vals
  , n.chains = n_chains
)

# update: 1K burnins
update(people_mcmc_model, 1000)

# fit model
people_mcmc_fit <- coda.samples(
  people_mcmc_model
  , variable.names = c(
    "deviance"
    , "pi"
    , "c"
    , "theta0", "theta1"
    , "alpha0", "beta0"
    , "alpha1", "beta1"
  )
  , n.iter = 40000
  , thin = 20
)

fits <- list()
fits$people_mcmc_fit <- people_mcmc_fit

First, we want to ensure that the model converged and chains are well-mixed. To do so, we inspect convergence of the deviance information criterion:

Convergence is achieved very quickly: the shrinkage factor is close to one already after iterations. Keeping only every 20th estimate helps to reduce autocorrelation substantially.

Posterior prevalence of people-centrism

Turning to the posterior density of \(\pi\), the prevalence of people-centrism in social media posts, it becomes immediately apparent that all three chains have converged on the reverse assignment, a problem resulting from the non-identifiability of the measurement model.³ Hence, I use post-hoc transformation to obtain the correct assignment.⁴

The posterior density of \(\pi\) is unimodal and, after post-hoc transformation, has its mean value 0.107. Importantly, with a total 3339 valid codings provided for 1489, the prevalence posterior density exhibits relatively little dispersion given that we have given the prevalence an uninformative (flat) Beta(1,1) prior density: 90% os posterior values are in the range 0.078, 0.143.

Posterior classification uncertainty

Turning to posterior classification uncertainties, we see that with only a few judgments per item, there is much variability in classification uncertainty when aggregating classifications across chains and iterations at the item level:⁵

While the majority of items can be assigned with little posterior classification uncertainty, there are items with both moderate (\(\text{SD}(c_i) \in [.25, .4)\)) and high (\(\text{SD}(c_i) \geq .4\)) levels of posterior classification uncertainty.⁶

Hence, for people-centrism in the validation items, we get the following mean and standard devation values of posterior classification uncertainty:

Posterior coder abilities

In addition to classification quality, we are also interested in the distribution of coder abilities.

First, we can inspect posterior estimates of coders’ sensitivity and specificity parameters:

The picture is relatively homogenous: Coders are generally found to be highly specific, that is, to perform well in correctly classifying negative items. The mass of most posterior densities of \(\theta_{0\cdot}\) parameters is in the range \([.75,1)\). With regard to coders’ true-positive detection abilities, there are some outliers with sensitivities in the range \([.4, .6]\) (e.g., coders 7-9, 17, 38, and 39) and even substantial posterior densitiy mass below .5 (specifically coder 32). Hence, the distribution of posterior means is more dispersed in case of sensitivities than specificities:

Having specified uninformative priors, the validation data gives reason to believe that the sampled coders are somewhat heterogenous in terms of classification abilities, at least with regard to sensitivities. This is confirmed when looking at the distributions of hyperparameters of sensitivity and specificity distributions:

Only \(\beta_0\), the second shape parameter of the specificity hyperdistribution, can be estimated with comparatively high precision. Take for instance the shape parameters of coders’ sensitivities, \(\alpha_1, \beta_1\). 80% of their values lie in the range \(\alpha_1 \in\) [1.493, 12.435] and \(\beta_1 \in\) [0.678, 6.827]. Due to the flexibility of the Beta-distribution into which these hyperparameters feed, depending on the selected quantile values we get differently shaped posterior densitie, as the next figure illustrates:

From the ability hyperdistributions we can conclude that the mass of coders are very specific and overwhelmingly non-adversarial but less perfect when classifying true-positive items.

(Dis)Agreement with majority-voting classifications

Most studies applying content analytical (i.e., human-coding based) instruments to measure populism in textual data use majority voting to aggregate codings at the item level. However, majority voting may produce biased results if coders provide noisy judgments. Hence, we want to know whether or not model-based posterior estimates and majority voting imply different classifications.

Indeed there are in total only 40 out of 1489 items (i.e., 2.686%) for which model-based and majority-voting classifications disagree. The vast share of this disagreement results from items that are classified as featuring people-centrism with majortiy voting but not when using BBA model-based aggregation (39). Importantly, this disagreement occurs most often where only one coder judged an item.

As a consequence of these differences, the empirical prevalence (not to confuse with \(\pi\)) differs somewhat between classification methods: 0.105 in case of majority voting vs. 0.08 in model-based classification.

Estimation

I obtain MCMC estimates using JAGS with three chains, 5K burn-in iterations, and 15K iterations with thinning parameter set to 15. These choices are based on inspecting convergence and autocorrelation in initial models with fewer iterations and less (or no) thinning.

Judging by the DIC, all chains mix nicely and converege quickly. And with the thinning parameter set to 15, we effectly reduce autocorrelation to tolerable levels.

As in the case of people-centrism classification, the model however converged on the inverse parameter assignment, so that we need to post-hoc transform estimates to the correct assignment.

Posterior prevalence of anti-elitism

The posterior density is unimodal and the mean of the prevalence is 0.282, that is, in expectation about every fifth social media post generated by party leaders or party accounts features anti-elitism.

Posterior classification uncertainty

We see that with only a few judgments per item, there is much variability in classification uncertainty when aggregating across chains and iterations at the item level:

While about one third of items can be assigned with little posterior classification uncertainty, a substantial number of items is characterized with moderate to high levels of posterior classification uncertainty (\(\text{SD}(c_i) \geq .25\)). Hence, we have the following mean and standard devation values of posterior classification uncertainty in anti-elitism classification:

Posterior coder abilities

Inspecting posterior estimates of coders’ sensitivity and specificity parameters, the picture is similar to that in case of people-centrism classification: Coders are generally highly specific, yet the samopled coders are more heterogenous with regard to their abilities to correctly classify positive items, as is illustrated by the following figure:

Having specified uninformative priors, the validation data gives reason to believe that the coder population is somewhat heterogenous in terms of classification abilities, but more often than not non-adversarial and better than chance. This is confirmed when looking at the distributions of hyperparameters of sensitivity and specificity distributions:

Compared to hyperparameter estimates in the case of people-centrism classification, densities are less dispersed with the minor exception of \(\alpha_0\) Take for instance the shape parameters of coders’ specificities, \(\alpha_1, \beta_1\). 80% of their values lie in the range \(\alpha_1 \in\) [1.133, 3.81] and \(\beta_1 \in\) [0.394, 1.413]. Due to the flexibility of the Beta-distribution into which these hyperparameters feed, depending on the selected quantile values we get differently shaped posterior densitie, as the next plot illustrates.

With above-median hyperparameter values, however, posterior ability distributions have their vast shares of mass on non-adversarial values (i.e., > .5), and again, we have reason to believe that coders are both highly specific as well as, though somewhat less so, sensitive.

(Dis)Agreement with majority-voting classifications

Again, we want to know whether or not model-based posterior estimates and majority voting imply different classifications.

Indeed there are in total only 29 out of 1489 items for which model-based and majority-voting classifications disagree. As a consequence of these differences, the empirical prevalence (not to confuse with \(\pi\)) differes only slightly between classification methods: 0.258 in case of majority voting vs. 0.271 in model-based classification.

Estimation

I obtain MCMC estimates using JAGS with three chains, 10K burn-in iterations, and 100K iterations with thinning parameter set to 50. These choices are based on inspecting convergence and autocorrelation in initial models with fewer iterations and less (or no) thinning.

Judging by the DIC, though chains tend to mix nicely, there is some downwards drift in DIC values that only levels off after the first 50K iterations. Hence, the shrinkage factor approaches one only after some ten thousand iterations. What is more, with the thinning parameter set to 50 we still have substantial autocorrelation. The estimates obtained by fitting the BBA model to the exclusionism judgments are thus to be taken with a grain of salt.

Not also that the model again converged on the inverse parameter assignment, so that I post-hoc transformed estimates to the correct assignment.

Posterior prevalence of exclusionism

The posterior density is unimodal, the mean of the prevalence is 0.096, and 90% of posterior estimates are in the range 0.073, 0.116.

Posterior classification uncertainty

We see that with only a few judgments per item, there already relaitively little classification uncertainty in most items:

The mass of items can be assigned with little posterior classification uncertainty, and there are only very few items with moderate to high levels of posterior classification uncertainty (\(\text{SD}(c_i) \geq .25\)). For exclusionism in the validation items, we get the following mean and standard devation values of posterior classification uncertainty:

Posterior coder abilities

In addition to classification quality, we are also interested in the distribution of coder abilities in exclusionism classification.

Inspecting posterior estimates of coders’ sensitivity and specificity parameters, we get a relatively clear-cut and familiar picture. Posterior estimates of both coders’ sensitivities and specificities are virutally all non-adversarial, and the mass of posterioir densities lies in regions that indicate better-than-chance classification abilities. Again, coders are somewhat more heterogenous wirth regard to true-positive classification abilities, as the following plot illustrates:

Given the validation data we have reason to believe that the coder population may be somewhat heterogenous in terms of true-positive classification abilities, whereas it is highly homogeneous in terms of true-negative classification abilities. This is supported when looking at the distributions of hyperparameters of sensitivity and specificity distributions:

With the minor exception of \(\beta_0\), which is characterized by a relatively tight credibility interval, densities are extremely dispersed. The resulting hyperdistributions, illustrated in the next figure, give reason to believe that the mass of the coder population is close to perfect in true-negative classification, and less perect, more heterogenous but overwhelmingly non-adversarial and better-than-chance in true-positive classification.

(Dis)Agreement with majority-voting classifications

Again, we want to know whether or not model-based posterior estimates and majority voting imply different classifications.

Indeed there are in total only 30 out of 1489 items for which model-based and majority-voting classifications disagree. All disagreement results from items that are classified as featuring exclusionism when using BBA model-based aggregation but not with majortiy voting. But as a consequence of random tie-breaking, the empirical prevalence still differs somewhat between classification methods: 0.065 in case of majority voting vs. 0.085 in model-based classification.